Section 1-3 - Parameter Tuning

In previous sections, we took the approach of using Scikit-learn as a black box. We now review how to tune the parameters of the model to make more accurate predictions.

Pandas - Extracting data



In [1]:

    
import pandas as pd
import numpy as np

df = pd.read_csv('../data/train.csv')

Pandas - Cleaning data



In [2]:

    
df = df.drop(['Name', 'Ticket', 'Cabin'], axis=1)

age_mean = df['Age'].mean()
df['Age'] = df['Age'].fillna(age_mean)

from scipy.stats import mode

mode_embarked = mode(df['Embarked'])[0][0]
df['Embarked'] = df['Embarked'].fillna(mode_embarked)

df['Gender'] = df['Sex'].map({'female': 0, 'male': 1}).astype(int)

pd.get_dummies(df['Embarked'], prefix='Embarked').head(10)
df = pd.concat([df, pd.get_dummies(df['Embarked'], prefix='Embarked')], axis=1)

df = df.drop(['Sex', 'Embarked'], axis=1)

cols = df.columns.tolist()
cols = [cols[1]] + cols[0:1] + cols[2:]

df = df[cols]

train_data = df.values

Scikit-learn - Training the model

The documentation for the Random Forest Classifier details the different input parameters of the model. These input parameters include the number of trees, and the number of branches each tree has. It is unclear, off-the-bat, which values would be optimal.

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

GridSearchCV allows us to test the desired range of input parameters, and review the performance of each set of values on a cross-validation basis. Here we review the number of features considered at each step a branch is made (max_features: 50% or 100% of features) and the maximum number of branches (max_depth: 5 levels or no limitations).



In [3]:

    
from sklearn.ensemble import RandomForestClassifier
from sklearn.grid_search import GridSearchCV

parameter_grid = {
    'max_features': [0.5, 1.],
    'max_depth': [5., None]
}

grid_search = GridSearchCV(RandomForestClassifier(n_estimators = 100), parameter_grid,
                            cv=5, verbose=3)



In [4]:

    
grid_search.fit(train_data[0:,2:], train_data[0:,0])









    



Fitting 5 folds for each of 4 candidates, totalling 20 fits
[GridSearchCV] max_features=0.5, max_depth=5.0 .................................
[GridSearchCV] ........ max_features=0.5, max_depth=5.0, score=0.843575 -   0.2s
[GridSearchCV] max_features=0.5, max_depth=5.0 .................................
[GridSearchCV] ........ max_features=0.5, max_depth=5.0, score=0.797753 -   0.2s
[GridSearchCV] max_features=0.5, max_depth=5.0 .................................
[GridSearchCV] ........ max_features=0.5, max_depth=5.0, score=0.797753 -   0.1s
[GridSearchCV] max_features=0.5, max_depth=5.0 .................................
[GridSearchCV] ........ max_features=0.5, max_depth=5.0, score=0.870787 -   0.1s
[GridSearchCV] max_features=0.5, max_depth=5.0 .................................
[GridSearchCV] ........ max_features=0.5, max_depth=5.0, score=0.831461 -   0.1s
[GridSearchCV] max_features=1.0, max_depth=5.0 .................................
[GridSearchCV] ........ max_features=1.0, max_depth=5.0, score=0.854749 -   0.2s
[GridSearchCV] max_features=1.0, max_depth=5.0 .................................
[GridSearchCV] ........ max_features=1.0, max_depth=5.0, score=0.797753 -   0.2s
[GridSearchCV] max_features=1.0, max_depth=5.0 .................................
[GridSearchCV] ........ max_features=1.0, max_depth=5.0, score=0.786517 -   0.2s
[GridSearchCV] max_features=1.0, max_depth=5.0 .................................
[GridSearchCV] ........ max_features=1.0, max_depth=5.0, score=0.842697 -   0.2s
[GridSearchCV] max_features=1.0, max_depth=5.0 .................................
[GridSearchCV] ........ max_features=1.0, max_depth=5.0, score=0.831461 -   0.2s
[GridSearchCV] max_features=0.5, max_depth=None ................................
[GridSearchCV] ....... max_features=0.5, max_depth=None, score=0.832402 -   0.2s
[GridSearchCV] max_features=0.5, max_depth=None ................................
[GridSearchCV] ....... max_features=0.5, max_depth=None, score=0.803371 -   0.2s
[GridSearchCV] max_features=0.5, max_depth=None ................................
[GridSearchCV] ....... max_features=0.5, max_depth=None, score=0.786517 -   0.2s
[GridSearchCV] max_features=0.5, max_depth=None ................................
[GridSearchCV] ....... max_features=0.5, max_depth=None, score=0.808989 -   0.2s
[GridSearchCV] max_features=0.5, max_depth=None ................................
[GridSearchCV] ....... max_features=0.5, max_depth=None, score=0.831461 -   0.2s
[GridSearchCV] max_features=1.0, max_depth=None ................................
[GridSearchCV] ....... max_features=1.0, max_depth=None, score=0.837989 -   0.3s
[GridSearchCV] max_features=1.0, max_depth=None ................................
[GridSearchCV] ....... max_features=1.0, max_depth=None, score=0.803371 -   0.3s
[GridSearchCV] max_features=1.0, max_depth=None ................................
[GridSearchCV] ....... max_features=1.0, max_depth=None, score=0.780899 -   0.4s
[GridSearchCV] max_features=1.0, max_depth=None ................................
[GridSearchCV] ....... max_features=1.0, max_depth=None, score=0.808989 -   0.3s
[GridSearchCV] max_features=1.0, max_depth=None ................................
[GridSearchCV] ....... max_features=1.0, max_depth=None, score=0.848315 -   0.3s





    



[Parallel(n_jobs=1)]: Done   1 jobs       | elapsed:    0.2s
[Parallel(n_jobs=1)]: Done  20 out of  20 | elapsed:    4.3s finished






    









    Out[4]:





GridSearchCV(cv=5,
       estimator=RandomForestClassifier(bootstrap=True, compute_importances=None,
            criterion='gini', max_depth=None, max_features='auto',
            min_density=None, min_samples_leaf=1, min_samples_split=2,
            n_estimators=100, n_jobs=1, oob_score=False, random_state=None,
            verbose=0),
       fit_params={}, iid=True, loss_func=None, n_jobs=1,
       param_grid={'max_features': [0.5, 1.0], 'max_depth': [5.0, None]},
       pre_dispatch='2*n_jobs', refit=True, score_func=None, scoring=None,
       verbose=3)

We now review the results.



In [5]:

    
grid_search.grid_scores_









    Out[5]:





[mean: 0.82828, std: 0.02798, params: {'max_features': 0.5, 'max_depth': 5.0},
 mean: 0.82267, std: 0.02621, params: {'max_features': 1.0, 'max_depth': 5.0},
 mean: 0.81257, std: 0.01747, params: {'max_features': 0.5, 'max_depth': None},
 mean: 0.81594, std: 0.02437, params: {'max_features': 1.0, 'max_depth': None}]

We sort the results, and determine the best-performing tuning parameters.



In [6]:

    
sorted(grid_search.grid_scores_, key=lambda x: x.mean_validation_score)
grid_search.best_score_
grid_search.best_params_









    Out[6]:





{'max_depth': 5.0, 'max_features': 0.5}

We then set these tuning parameters to our model.



In [7]:

    
model = RandomForestClassifier(n_estimators = 100, max_features=0.5, max_depth=5.0)
model = model.fit(train_data[0:,2:],train_data[0:,0])

Exercise

Write the code so that grid_search refits model with the best tuning parameters to the entire dataset after these parameters are found, and hence allow us to skip the two lines of code above.

Scikit-learn - Making predictions



In [8]:

    
df_test = pd.read_csv('../data/test.csv')

df_test = df_test.drop(['Name', 'Ticket', 'Cabin'], axis=1)

df_test['Age'] = df_test['Age'].fillna(age_mean)

fare_means = df.pivot_table('Fare', index='Pclass', aggfunc='mean')
df_test['Fare'] = df_test[['Fare', 'Pclass']].apply(lambda x:
                            fare_means[x['Pclass']] if pd.isnull(x['Fare'])
                            else x['Fare'], axis=1)

df_test['Gender'] = df_test['Sex'].map({'female': 0, 'male': 1}).astype(int)
df_test = pd.concat([df_test, pd.get_dummies(df_test['Embarked'], prefix='Embarked')],
                axis=1)

df_test = df_test.drop(['Sex', 'Embarked'], axis=1)

test_data = df_test.values

output = model.predict(test_data[:,1:])









    



/Users/savarin/anaconda/lib/python2.7/site-packages/pandas/core/index.py:496: FutureWarning: scalar indexers for index type Int64Index should be integers and not floating point
  type(self).__name__),FutureWarning)

Pandas - Preparing for submission



In [9]:

    
result = np.c_[test_data[:,0].astype(int), output.astype(int)]


df_result = pd.DataFrame(result[:,0:2], columns=['PassengerId', 'Survived'])
df_result.to_csv('../results/titanic_1-3.csv', index=False)